PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types
of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in
PKCS#11 mode using OpenSC to handle system login. libpam-pkcs11 is a pretty
easy to use module that will let you log into your system locally using a
PKCS#11 token locally.
One of the least documented things, though, was how to use an OpenSC PKCS#11
token in Chrome. First, close all web browsers you have open.
About a year ago, I bought a Projector after I finally admitted that I could
actually use a TV in my apartment. I settled on buying a
ViewSonic PJD5132.
It was a really great value, and has been nothing short of a delight to own.
I was always a bit curious about the DB9 connector on the back of the unit,
so I dug into the user manual, and found some hex code strings in there. So,
last year, between my last gig at the
Sunlight Foundtion and
USDS, I spent some time wandering around the US,
hitting up DebConf, and exploring Washington
DC. Between trips, I set out to figure out exactly what was going on with my
Projector, and see if I could make it do anything fun.
So, I started off with basics, and tried to work out how these command codes
were structured. I had a few working codes, but to write clean code, I'd be
better off understanding why the codes looked like they do. Let's look at
the "Power On" code.
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D
Some were 10 bytes, other were 11, and most started with similar looking
things. The first byte was usually a 0x06 or 0x07, followed by two
bytes 0x14 0x00, and either a 0x04 or 0x05. Since the first few bytes
were similarly structured, I assumed the first octet (either 0x06 or 0x07)
was actually a length, since the first 4 octets seemed always present.
So, my best guess is that we have a Length byte at index 0, followed by
two bytes for the Protocol, a flag for if you're Reading or Writing (best
guess on that one), and opaque data following that. Sometimes it's a const
of sorts, and sometimes an octet (either little or big endian, confusingly).
Right. OK. So, let's get to work. In the spirit of code is data, data is code,
I hacked up some of the projector codes into a s-expression we can use later.
The structure of this is boring, but it'll let us both store the command
code to issue, as well as define the handler to read the data back.
(setv*commands*; function type family control'((power-onnilnil(0x060x140x000x040x000x340x110x000x000x5D))(power-offnilnil(0x060x140x000x040x000x340x110x010x000x5E))(power-statusconstpower(0x070x140x000x050x000x340x000x000x110x000x5E))(resetnilnil(0x060x140x000x040x000x340x110x020x000x5F))...
As well as defining some of the const responses that come back from the
Projector itself. These are pretty boring, but it's helpful to put a
name to the int that falls out.
After defining a few simple functions to write the byte arrays to the serial
port as well as reading and understanding responses from the projector, I could
start elaborating on some higher order functions that can talk projector. So
the first thing I wrote was to make a function that converts the command
entry into a native Hy function.
Cool. So, now we can import things like power-on from *commands* which
takes a single argument (serial) for the serial port, and it'll send a
command, and return the response. The best part about all this is you only
have to define the data once in a list, and the rest comes for free.
Finally, I do want to be able to turn my projector on and off over the network
so I went ahead and make a Flask "API" on top of this. First, let's define
a macro to define Flask routes:
I ll be at HOPE 11 this year - if anyone else will be around, feel free to send me an email! I won t have a phone on me (so texting only works if you use Signal!)
Looking forward for a chance to see everyone soon!
Around a year ago, I started hacking together a machine readable version
of the OSI approved licenses list, and casually picking parts up until it
was ready to launch. A few weeks ago, we officially announced
the osi license api, which is now
live at api.opensource.org.
I also took a whack at writing a few API bindings, in
Python,
Ruby,
and using the models from the API implementation itself in
Go. In the following
few weeks, Clint wrote one in Haskell,
Eriol wrote one in Rust,
and Oliver wrote one in R.
The data is sourced from a repo on GitHub,
the licenses repo under OpenSourceOrg. Pull Requests against that repo are
wildly encouraged! Additional data ideas, cleanup or more hand collected data
would be wonderful!
In the meantime, use-cases for using this API range from language package
managers pulling OSI approval of a licence programatically to using a license
identifier as defined in one dataset (SPDX, for exampele), and using that
to find the identifer as it exists in another system (DEP5, Wikipedia,
TL;DR Legal).
Patches are hugly welcome, as are bug reports or ideas! I'd also love more
API wrappers for other languages!
A while back, I found myself in need of two webservers that would terminate TLS (with different rules). I wanted to run some custom code I d written (which uses TLS peer authentication), and also nginx on port 443.
The best way I figured out how to do this was to write a tool to sit on port 443, and parse TLS Client Hello packets, and dispatch to the correct backend depending on the SNI name.
SNI, or Server Name Indication allows the client to announce (yes over cleartext!) what server it s looking for, similar to the HTTP Host header. Sometimes, like in the case above, the Host header won t work, since you ve already done a TLS handshake by the time you figure out who they re looking for.
I also spun the Client Hello parser out into its own importable package, just in case someone else finds themselves in this same boat.
The code s up on github.com/paultag/sniff!
Last week, I posted about python-sense,
and API wrapper for the internal Sense API. I wrote this so that I could
pull data about myself into my own databases, allowing me to use that
information for myself.
One way I'm doing this is by pulling my room data into an
InfluxDB database, letting me run time series
queries against my environmental data.
I'm able to run this on a cron, automatically loading data from the Sense
API into my Influx database. I can then use that with something like
Grafana, to check out what my room looks like over
time.
A while back, I saw a Kickstarter
for one of the most well designed and pretty sleep trackers on the market. I
fell in love with it, and it has stuck with me since.
A few months ago, I finally got my hands on one and started to track my data.
Naturally, I now want to store this new data with the rest of the data I have
on myself in my own databases.
I went in search of an API, but I found that the Sense API hasn't been published
yet, and is being worked on by the team. Here's hoping it'll land soon!
After some subdomain guessing, I hit on api.hello.is.
So, naturally, I went to take a quick look at their Android app and network
traffic, lo and behold, there was a pretty nicely designed API.
This API is clearly an internal API, and as such, it's something that
should not be considered stable. However, I'm OK with a fragile API,
so I've published a quick and dirty API wrapper for the Sense API
to my GitHub..
I've published it because I've found it useful, but I can't promise the world,
(since I'm not a member of the Sense team at Hello!), so here are a few ground
rules of this wrapper:
I make no claims to the stability or completeness.
I have no documentation or assurances.
I will not provide the client secret and ID. You'll have to find them on
your own.
This may stop working without any notice, and there may even be really nasty
bugs that result in your alarm going off at 4 AM.
Send PRs! This is a side-project for me.
This module is currently Python 3 only. If someone really needs Python 2
support, I'm open to minimally invasive patches to the codebase using
six to support Python 2.7.
Working with the API:
First, let's go ahead and log in using python -m sense.
$ python -m sense
Sense OAuth Client ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense OAuth Client Secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense email: paultag@gmail.com
Sense password:
Attempting to log into Sense's APISuccess!Attempting to query the Sense APIThe humidity is **just right**.The air quality is **just right**.The light level is **just right**.It's **pretty hot** in here.
The noise level is **just right**.
Success!
Now, let's see if we can pull up information on my Sense:
>>>fromsenseimportSense>>>sense=Sense()>>>sense.devices()'senses':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'11a1','last_updated':1466991060000,'state':'NORMAL','wifi_info':'rssi':0,'ssid':'Pretty Fly for a WiFi (2.4 GhZ)','condition':'GOOD','last_updated':1462927722000 ,'color':'BLACK' ],'pills':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'2','last_updated':1466990339000,'battery_level':87,'color':'BLUE','state':'NORMAL' ]
Neat! Pretty cool. Look, you can even see my WiFi AP! Let's try some more
and pull some trends out.
I plan to keep maintaining it as long as it's needed, so I welcome
co-maintainers, and I'd love to see what people build with it! So far, I'm
using it to dump my room data into InfluxDB, pulling information on my room
into Grafana. Hopefully more to come!
Happy hacking!
As some of the world knows full well by now, I've been noodling with Go
for a few years, working through its pros, its cons, and thinking a lot
about how humans use code to express thoughts and ideas. Go's got a lot of
neat use cases, suited to particular problems, and used in the right place,
you can see some clear massive wins.
I've started writing Debian tooling in Go, because it's a pretty natural fit.
Go's fairly tight, and overhead shouldn't be taken up by your operating system.
After a while, I wound up hitting the usual blockers, and started to build up
abstractions. They became pretty darn useful, so, this blog post is announcing
(a still incomplete, year old and perhaps API changing) Debian package for Go.
The Go importable name is pault.ag/go/debian. This contains a lot of utilities
for dealing with Debian packages, and will become an edited down "toolbelt"
for working with or on Debian packages.
Module Overview
Currently, the package contains 4 major sub packages. They're a changelog
parser, a control file parser, deb file format parser, dependency parser
and a version parser. Together, these are a set of powerful building blocks
which can be used together to create higher order systems with reliable
understandings of the world.
changelog
The first (and perhaps most incomplete and least tested) is a changelog file
parser.. This provides the
programmer with the ability to pull out the suite being targeted in the
changelog, when each upload was, and the version for each. For example, let's
look at how we can pull when all the uploads of Docker to sid took place:
funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")iferr!=nilpanic(err)allEntries,err:=changelog.Parse(resp.Body)iferr!=nilpanic(err)for_,entry:=rangeallEntriesfmt.Printf("Version %s was uploaded on %s\n",entry.Version,entry.When)
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
control
Next is one of the most complex, and one of the oldest parts of go-debian,
which is the control file parser
(otherwise sometimes known as deb822). This module was inspired by the way
that the json module works in Go, allowing for files to be defined in code
with a struct. This tends to be a bit more declarative, but also winds up
putting logic into struct tags, which can be a nasty anti-pattern if used too
much.
The first primitive in this module is the concept of a Paragraph, a struct
containing two values, the order of keys seen, and a map of string to string.
All higher order functions dealing with control files will go through this
type, which is a helpful interchange format to be aware of. All parsing of
meaning from the Control file happens when the Paragraph is unpacked into
a struct using reflection.
The idea behind this strategy that you define your struct, and let the Control
parser handle unpacking the data from the IO into your container, letting you
maintain type safety, since you never have to read and cast, the conversion
will handle this, and return an Unmarshaling error in the event of failure.
Additionally, Structs that define an anonymous member of control.Paragraph
will have the raw Paragraph struct of the underlying file, allowing the
programmer to handle dynamic tags (such as X-Foo), or at least, letting
them survive the round-trip through go.
The default decoder
contains an argument, the ability to verify the input control file using an
OpenPGP keyring, which is exposed to the programmer through the
(*Decoder).Signer() function. If the passed argument is nil, it will not
check the input file signature (at all!), and if it has been passed, any
signed data must be found or an error will fall out of the NewDecoder call.
On the way out, the opposite happens, where the struct is introspected,
turned into a control.Paragraph, and then written out to the io.Writer.
Here's a quick (and VERY dirty) example showing the basics of reading and
writing Debian Control files with go-debian.
packagemainimport("fmt""io""net/http""strings""pault.ag/go/debian/control")typeAllowedPackagestructPackagestringFingerprintstringfunc(a*AllowedPackage)UnmarshalControl(instring)errorin=strings.TrimSpace(in)chunks:=strings.SplitN(in," ",2)iflen(chunks)!=2returnfmt.Errorf("Syntax sucks: '%s'",in)a.Package=chunks[0]a.Fingerprint=chunks[1][1:len(chunks[1])-1]returnniltypeDMUAstructFingerprintstringUidstringAllowedPackages[]AllowedPackage control:"Allow" delim:"," funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/dm.txt")iferr!=nilpanic(err)decoder,err:=control.NewDecoder(resp.Body,nil)iferr!=nilpanic(err)fordmua:=DMUAiferr:=decoder.Decode(&dmua);err!=niliferr==io.EOFbreakpanic(err)fmt.Printf("The DM %s is allowed to upload:\n",dmua.Uid)for_,allowedPackage:=rangedmua.AllowedPackagesfmt.Printf(" %s [granted by %s]\n",allowedPackage.Package,allowedPackage.Fingerprint)
Output (truncated!) looks a bit like:
...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM <aelmahmoudy@sabily.org> is allowed to upload:
covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
...
...
deb
Next up, we've got the deb module. This contains code to handle reading
Debian 2.0 .deb files. It contains a wrapper that will parse the control
member, and provide the data member through the
archive/tar interface.
Here's an example of how to read a .deb file, access some metadata, and
iterate over the tar archive, and print the filenames of each of the
entries.
dependency
The dependency package provides an interface to parse and compute
dependencies. This package is a bit odd in that, well, there's no other
library that does this. The issue is that there are actually two different
parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev)
and another in C (in dpkg).
To date, this has resulted in me filing
threedifferentbugs.
I also found a broken package in the
archive,
which actually resulted in another bug being (totally accidentally)
already fixed.
I hope to continue to run the archive through my parser in hopes of finding
more bugs! This package is a bit complex, but it basically just returns what
amounts to be an AST
for our Dependency lines. I'm positive there are bugs, so file them!
funcmain()dep,err:=dependency.Parse("foo bar, baz, foobar [amd64] bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")iferr!=nilpanic(err)anySparc,err:=dependency.ParseArch("sparc")iferr!=nilpanic(err)for_,possi:=rangedep.GetPossibilities(*anySparc)fmt.Printf("%s (%s)\n",possi.Name,possi.Arch)
Gives the output:
foo (<nil>)
baz (<nil>)
fnord (armhf)
version
Right off the bat, I'd like to thank
Michael Stapelberg for letting me graft this
out of dcs and into the go-debian package.
This was nearly entirely his work (with a one or two line function I added
later), and was amazingly helpful to have. Thank you!
This module implements Debian version comparisons and parsing, allowing for
sorting in lists, checking to see if it's native or not, and letting the
programmer to implement smart(er!) logic based on upstream (or Debian)
version numbers.
This module is extremely easy to use and very straightforward, and not worth
writing an example for.
Final thoughts
This is more of a "Yeah, OK, this has been useful enough to me at this point
that I'm going to support this" rather than a "It's stable!" or even
"It's alive!" post. Hopefully folks can report bugs and help iterate on
this module until we have some really clean building blocks to build
solid higher level systems on top of. Being able to have multiple libraries
interoperate by relying on go-debian will be a massive ease.
I'm in need of more documentation, and to finalize some parts of the older
sub package APIs, but I'm hoping to be at a "1.0" real soon now.
As nearly anyone who's worked with me will attest to, I've long since
touted nedbat's talk
Pragmatic Unicode, or, How do I stop the pain?
as one of the most foundational talks, and required watching for all programmers.
The reason is because netbat hits on something bigger - something more
fundamental than how to handle Unicode -- it's how to handle data which is
relative.
For those who want the TL;DR, the argument is as follows:
Facts of Life:
Computers work with Bytes. Bytes go in, Bytes go out.
Relative Data
I've started to think more about why we do the things we do when we write
code, and one thing that continues to be a source of morbid schadenfreude
is watching code break by failing to handle Unicode right. It's hard! However,
watching what breaks lets you gain a bit of insight into how the author
thinks, and what assumptions they make.
When you send someone Unicode, there are a lot of assumptions that have to be
made. Your computer has to trust what you (yes, you!) entered into your web
browser, your web browser has to pass that on over the network (most of the
time without encoding information), to a server which reads that bytestream,
and makes a wild guess at what it should be. That server might save it to a
database, and interpolate it into an HTML template in a different encoding
(called Mojibake), resulting
in a bad time for everyone involved.
Everything's awful, and the fact our computers can continue to display
text to us is a goddamn miracle. Never forget that.
When it comes down to it, when I see a byte sitting on a page, I don't know
(and can't know!) if it's Windows-1252, UTF-8, Latin-1, or EBCDIC. What's
a poem to me is terminal garbage to you.
Over the years, hacks have evolved. We have
magic numbers,
and plain ole' hacks to just guess based on the content. Of course, like
all good computer programs, this has lead to its fair share of hilarious
bugs,
and there's nothing stopping files from (validly!) being multiple things at the
same time.
Like many things, it's all in the eye of the beholder.
Timezones
Just like Unicode, this is a word that can put your friendly neighborhood
programmer into a series of profanity laden tirades. Go find one in the wild,
and ask them about what they think about timezone handling bugs they've seen.
I'll wait. Go ahead.
Rants are funny things. They're fun to watch. Hilarious to give. Sometimes
just getting it all out can help. They can tell you a lot about the true
nature of problems.
It's funny to consider the isomorphic nature of Unicode rants and Timezone
rants.
I don't think this is an accident.
U n i c o d e timezone Sandwich
Ned's Unicode Sandwich applies -- As early as we can, in the lowest level
we can (reading from the database, filesystem, wherever!), all datetimes
must be timezone qualified with their correct timezone. Always. If you mean
UTC, say it's in UTC.
Treat any unqualified datetimes as "bytes". They're not to be trusted.
Never, never, never trust 'em. Don't
process any datetimes until you're sure they're in the right timezone.
This lets the delicious inside of your datetime sandwich handle timezones
with grace, and finally, as late as you can, turn it back into bytes
(if at all!). Treat locations as tzdb entries, and qualify datetime
objects into their absolute timezone (EST, EDT, PST, PDT)
It's not until you want to show the datetime to the user again should you
consider how to re-encode your datetime to bytes. You should think about
what flavor of bytes, what encoding -- what timezone -- should I be
encoding into?
TEST
Just like Unicode, testing that your code works with datetimes is important.
Every time I think about how to go about doing this, I think about that
one time that mjg59 couldn't book a flight
starting Tuesday from AKL, landing in HNL on Monday night, because
United couldn't book the last leg to SFO. Do you ever assume dates only go
forward as time goes on? Remember timezones.
Construct test data, make sure someone in New Zealand's
+13:45 can correctly talk with
their friends in
Baker Island's -12:00,
and that the events sort right.
Just because it's Noon on New Years Eve in England doesn't mean it's not
1 AM the next year in New Zealand. Places a few miles apart may go on Daylight
savings different days. Indian Standard Time is not even aligned on the hour
to GMT (+05:30)!
Test early, and test often. Memorize a few timezones, and challenge
your assumptions when writing code that has to do with time. Don't use
wall clocks to mean monotonic time. Remember there's a whole world out there,
and we only deal with part of it.
It's also worth remembering, as Andrew Pendleton
pointed out to me, that it's possible that a datetime isn't even unique for a
place, since you can never know if 2016-11-06 01:00:00 in America/New_York
(in the tzdb) is the first one, or second one. Storing EST or EDT along
with your datetime may help, though!
Pitfalls
Improper handling of timezones can lead to some interesting things, and failing
to be explicit (or at least, very rigid) in what you expect will lead to an
unholy class of bugs we've all come to hate. At best, you have confused
users doing math, at worst, someone misses a critical event, or our
security code fails.
I recently found what I regard to be a pretty bad
bug in apt (which David has prepared a
fix
for and is pending upload, yay! Thank you!), which boiled down to documentation
and code expecting datetimes in a timezone, but accepting any timezone, and
silently treating it as UTC.
The solution is to hard-fail, which is an interesting choice to me (as a vocal
fan of timezone aware code), but at the least it won't fail by
misunderstanding what the server is trying to communicate, and I do understand
and empathize with the situation the apt maintainers are in.
Final Thoughts
Overall, my main point is although most modern developers know how to deal
with Unicode pain, I think there is a more general lesson to learn -- namely,
you should always know what data you have, and always remember what it is.
Understand assumptions as early as you can, and always store them with the data.
Back in 2014, Mako ran a Boston Iron Blogger chapter, where you had to blog once a week, or you owed $5 into the pot. A while later, I ran it (along with Molly and Johns), and things were great.
When I moved to DC, I had already talked with Tom Lee and Eric Mill about running a DC Iron Blogger chapter, but it hasn t happened in the year and a half I ve been in DC.
This week, I make good on that, with a fantastic group set up at dc.iron-blogger.com; with more to come (I m sure!).
Looking forward to many parties and though provoking blog posts in my future. I m also quite pleased I ll be resuming my blogging. Hi, again, planet Debian!
There s a lot of things I d like to blog about. The last version of OpenStack, the OpenStack Liberty design summit, Kilo in the official jessie-backports repositories, etc. Maybe the most interesting part of this blog post is the last bit at the end, about a major change in the packaging workflow for OpenStack in Debian. Please read on
OpenStack release names reminder
Just a reminder to make it easier for the average Debian reader who may know Debian well, but not OpenStack.
OpenStack 2014.1, is Icehouse, and is the version in Jessie. 2014.2 is Juno and was released right before the freeze of Jessie. 2015.1.0 is what has been released just right after jessie, on the 30th of April. Liberty, which probably will be called 12 (as this will be the 12th release of OpenStack), and not 2015.2 (this has been discussed in Vancouver), will be released in about 5 months form now.
The last summit, in Vancouver, BC, Canada, was the Liberty summit, as the OpenStack conventions are always named after the next release (since we are discussing what we will be doing during the next development cycle).
OpenStack 2015.1.0, aka Kilo, release in Debian
5 days after the release of Jessie, OpenStack 2015.1.0, aka Kilo, was released. Since I couldn t upload to unstable during the freeze, I was holding a lot of packages, and when I did upload them, there was about 20 packages of mine in the FTP master s NEW queue. Though, since the DSA want to use OpenStack for the Debian infrastructure, the 20 packages were fast track into Sid, thanks to the work of Paultag (thanks man!).
OpenStack Kilo in the official Jessie backports
Previously, I was only uploading OpenStack packages to Debian unstable, and maintaining a non-official Debian repositories for backports to Debian stable. However, for multiple reasons, this wasn t satisfying.
Then, after packages migrated to Stretch, I started to upload to Debian backports. And right before the summit, almost everything went in. Only python-pysaml2 was missing (as I discovered too late that version 2.0.0 breaks Keystone which needs version 2.4.0). In fact, the last bits of the Kilo release reached jessie-backports in the middle of the OpenStack Liberty summit.
Removal of the Debian install-guide from the official site
As there was not enough efforts working on the documentation, unfortunately, the link to the Debian install-guide has been removed from docs.openstack.org. IMO, this is mostly due to a bad communication between myself and the doc team, and also because one person who promised to work on the Debian side of the install-guide failed to warn everyone that he finally couldn t (as his managers assigned him to something else).
I hope this will soon be reverted. During the Vancouver summit, I had the opportunity to discuss with the doc team about re-inclusion of the Debian install-guide. Unfortunately, as they are moving away from the XML source format to a more standard RST-based system, the current documentation is frozen, so it seems more realistic to hold on until all of the install-guide is switched to RST.
OpenStack Debian image listed on apps.openstack.org
There s a new area on the openstack.org where images and apps for OpenStack are listed. Under the glance image tab, you will see that both the Jessie and the weekly testing image are listed. There s also a nice, easily identifiable Debian logo to link to these images.
Also, as there are trademark problems with the Ubuntu images which makes them harder to redistribute, the Murano project (which is shipping a system to automatically install apps that to installed within a few clicks on an OpenStack cloud) decided to switch to Debian for their base image.
Debian listed in the OpenStack market place
On the openstack.org site, there s a section called Marketplace. In there, vendors supporting OpenStack are listed. To get there, a vendor needs to 1/ have a defined set of OpenStack project supported by the distribution (Debian already has a way more than the required set), 2/ sign some kind of agreement with the OpenStack foundation, and 3/ pay some sponsoring money. During the summit, I discussed this with Jonathan Bryce, from the OpenStack foundation, and he agreed that Debian would not have to pay for this (since we aren t a big company with big money).
I have put Jonathan and Neil (our Debian Project Leader) in touch so that signing the document may happen, though since we were all busy with the summit, I do not expect Jonathan to send the documents right away. Hopefully, this will be fixed before the end of this month of May 2015.
Debian (and Ubuntu) packages collaboratively maintained upstream
Since about forever (forever is 5 years in the OpenStack world ), I pushed for more collaboration on OpenStack packaging between Debian package maintainers and Canonical. However, for some reasons which I do not wish to expand on in this blog post, it has been socially hard to do so. Also, Canonical always used BZR, which wasn t to the tastes of everyone.
But during the Liberty summit, some very good things happened. First of all, Launchpad is now able to support Git (it s been a few weeks it does in fact). Even though it will take a bit of time before the Canonical server team switches to it, we can consider that this problem is already out of the way.
Then it looks like Canonical are now more open than before for collaboration with Debian on the OpenStack packaging. Note that we actually did some work together already, but now we both would like a full alignment of *all* of our packages.
I have discussed this with James Page, who is the head of Canonical s server team. We will first start to do so on the dependencies: this includes all of the python-*client libraries, but also all of python-oslo.* (the Oslo libs are use by all of the projects and are kind of unifying the project), plus all the third party dependencies the project relies on. James already pushed new versions of some Oslo libraries to Experimental (in order to not overwrite Kilo), which are adding transition packages needed for Ubuntu. We wont need those in Debian, but we want to welcome them to keep the same source packages.
We will then later try to merge the core projects if we can. Unfortunately, since the packaging of the core projects (ie: Nova, Neutron, Cinder, Glance, etc.) was forked, merging probably will be a bit painful. We will have to make some decisions on how this happen. I am however confident that it will be done during the Liberty release cycle.
Move of the packaging to upstream Gerrit
A few weeks after the summit, I wrote a proposal to upstream OpenStack dev list, with as subject: Adding packaging as an OpenStack project . What it means is that I have proposed to have Debian/Ubuntu packaging to happen in upstream infrastructure, using Gerrit, and building packages using upstream cloud. We will add all the tests we can, like building with unit tests, lintian, piuparts, adequate, but also maybe a full installation of the packages with functional tests.
My proposal is here:
http://lists.openstack.org/pipermail/openstack-dev/2015-May/064848.html
As everything, this translates into a Gerrit review process:
https://review.openstack.org/#/c/185187/
As you can read in the above thread, Fedora/RDO people, which have used a Gerrit work-flow for a long time already, also would like to join. But it looks like we ll be doing 2 teams: one for RPMs and one for debs.
The proposal is currently under review by the OpenStack technical committee, which will accept (or not) if the packaging project can be fully considered as an OpenStack project. I expect a final answer next Tuesday. Note that if they deny, we can still use the stackforge namespace instead, their decision is just about the TC blessing the project as being OpenStack or not.
What s very nice about this, is that not only we will have a better collaboration between Debian & Ubuntu, better automated testing and Q/A, this also opens contributions to potentially anyone. Especially, we welcome operation people, those who are doing actual big deployments.
Sure, it was possible before, but I often had the feedback that many were scared to break anything when trying to contribute. Thanks to the CI/CD form upstream infra, and the Gerrit peer review process, it wont be a problem anymore. So we do expect operation people to contribute more. I will also push more upstream packaging within Mirantis, so that MOS (Mirantis OpenStack) aligns fully with Debian & Ubuntu as well.
Another good thing, is that it will be easier for the puppet team to support Debian (they historically were more Ubuntu oriented), and it s going to be super easy for them to request for packaging fixes. I hope we will be able to work hand-to-hand with them, adding puppet deployment checks in the packaging repo, and packaged deployments within the puppet Gerrit review process.
Assembly
Combine Soylent and Simple Syrup. Create what I m going to start to call Soylent Syrup . Enjoy that one, folks.
Add ice to a rocks glass, pour Soylent Syrup over ice. Add Sherry, Vermouth and Campari. Stir. Garnish with an orange twist.
Big thanks to Matthew Garrett for sparking this one.
When paultag recently announced a project to try to move debian infrastructure to python3, my first thought was how large that undertaking would likely be. It seems like a classic engineering task, full of work and nit-picky details to get right, useful/necessary in the long-term, painful in the short-term, and if you manage to pull it off successfully, the best you can usually hope for is that no one will notice that it was done at all. I always find that kind of task a little off-putting and difficult to tackle, but I was happy to see someone driving the project, since it does need to get done. Debian is potentially also in a position to help the upstream python community, because we have a pretty good view of what things are being used, at least within our own ecosystem. I'm happy to say that i also missed one of the other great benefits of paultag's audacious proposal, which is how it has engaged people who already knew about debian but who aren't yet involved. Evidence of this engagement is already visible on the py3porters-devel mailing list. But if that wasn't enough, I ran into a friend recently who told me, "Hey, I found a way to contribute to debian finally!" and pointed me to the py3-porters project. People want to contribute to the project, and are looking for ways in. So cheers to the people who propose audacious projects and make them inviting to everyone, newcomers included. And cheers to the people who step up to potentially daunting work, stake out a task, roll up their sleeves, and pitch in. Even if the py3porters project doesn't move all of debian's python infrastructure to pyt3 as fast as paultag wants it to, i think it's already a win for the project as a whole. I am looking forward to seeing what comes out of it (and it's reminding me i need to port some of my own python work, too!) The next time you stumble over something big that needs doing in debian, even something that might seem impossible, please make it inviting, and dive in. The rest of the project will grow and improve from the attempt.Tags: py3-porters
The percentage that women in Debian occupy as DDs is ~2%. Yes, just ~2% ladies that are DDs! So that means ~98% of DDs are gentelmen.
I know there are more of ladies in Debian, so I firstly urge you, for love of Debian, to apply if you are contributing to this project, love its community and want to see Debian taking over the universe (okay, it seems that we conquered outer space so we need a help on Earth).
So why is the number this low? Well maybe it's too precious to us currently inside that we want to prevent it being spoiled from outside. Also there seems to be not that much of younger DDs. Why is that important - well, young people like to do it and not to think about it. Many time they just break it, but many time they also do a breakthrough. Why is difference important and why should we embrace it? It's very important because it breaks a monopoly on view and behavior. It brings views not just from a larger number of people, but also from people from different backgrounds, and in constructive conversation it can put even more pluses on current workflow or it can counter it with good arguments. In a project of its size and worldwide geolocation of its developers, this is true for Debian more then any other projects I know. We need more women so we can balance our inner workings and have a better understanding of humanity and how is it moving, what and why does it need and where is it steering. That way we can produce a community which will improve quality of OS that we produce - because of sheer number of different people working on the same thing bringing to it its own personal touch. So, ladies and youth all over the world, unite and join in Debian because without diversity Debian can't grow beyond its current size. Also, no, Debian is not about code only, it needs painters, musicians, people that want to talk about Debian, people that share love and happiness, people that want to build better communities, UI/UX designers, makers, people who know how to repair a bike, athletes, homebrew beer producers, lawyers (just while world gets rid of laws, then we don't need you), actors, writters... Why, well because world and communities are made up from all that diversity and that's what makes it a better and not a monotone place.
But I just use Debian. Well, do you feel love towards Debian and its work? Would you like to feel more as integral part of community? If the answer is big fat YES, then you should be a DD too. Every person that feels it's part of Debians philosophy about freedom and behaving in good manner should join Debian. Every person that feels touched and enhanced by Debian's work should become part of community and share its experience how Debian touched their soul, impacted their life. If you love Debian, you should be free to contribute to it in whatever manner and you should be free to express your love towards it. If you think lintian is sexy, or shebang is a good friends of yours, or you enjoy talking to MadameZou about Debian and zombies (yeah, we do have all kinds of here), or you like Krita, or you hate the look of default XFCE theme, or you can prove that you a more crazy developer then paultag - just hop into community and try to integrate in it. You will meet great folks, have a lot of conversation about wine and cheese, play some dangerous card games and even learn about things like bokononism (yeah I am looking at you dkg!).
Now for the current Debian community - what the hell is packaging and non-packaging Debian Developer? Are one better then others? Do others stink? They don't know to hug? WHAT? Yes I know that inexperienced person shouldn't have a permission to access Debian packaging infrastructure, but I have the feeling that even that person knows that. Every person should have a place in Debian and acknowledge other fields. So yes, software developers need access to Debian packaging infrastructure, painters don't. I think we can agree on this. So lets abolish the stupid term and remove the difference in our community. Lets embrace the difference, because if someone writes a good poem about Debian heroism I could like it more then flashplugin-nonfree! Yep, I made that comparison on purpose so you can give a thought about it.
Debian has excellent community regarding operating system that it's producing. And it's not going away, not at least anytime soon. But it will not go forward if we don't give additional push as human beings, as people who care about their fellow Debianites. And we do care, I know that, we just need to push it more public. We don't hide bugs, we for sure shouldn't hide features. It will probably bring bad seeds too, but we have mechanisms and will to counter that. If we, on average 10 bad seeds, get some crazy good hacker or crazy lovely positive person like this lady, we will be on right path. Debian is a better place, it should lead in effort to bring more people into FLOSS world and it should allow people to bring more of diversity into Debian.
Hello, World
Been a while since my last blog post - things have been a bit hectic lately, and I ve not really had the time.
Now that things have settled down a bit I m in DC! I ve moved down south to join the rest of my colleagues at Sunlight to head up our State & Local team.
Leaving behind the brilliant Free Software community in Boston won t be easy, but I m hoping to find a similar community here in DC.
For the tl;dr: Docker FDW is a thing.
Star it, hack it, try it out. File bugs, be happy. If you want to see what it's
like to read, there's some example SQL down below.
The question is first, what the heck is a PostgreSQL Foreign Data Wrapper?
PostgreSQL Foreign Data Wrappers are plugins that allow C libraries
to provide an adaptor for PostgreSQL to talk to an external database.
Some folks have used this to wrap stuff like
MongoDB, which I always found
to be hilarous (and an epic hack).
Enter Multicorn
During my time at PyGotham, I saw a talk from
Wes Chow about something called
Multicorn. He was showing off some really neat
plugins, such as the git revision history of CPython, and parsed logfiles
from some stuff over at Chartbeat. This basically blew my mind.
All throughout the talk I was coming up with all sorts of things that I wanted
to do -- this whole library is basically exactly what I've been dreaming
about for years. I've always wanted to provide a SQL-like interface
into querying API data, joining data cross-API using common crosswalks,
such as using Capitol Words to query for
Legislators, and use the
bioguide ids
to JOIN against the congress api
to get their Twitter account names.
My first shot was to Multicorn the new
Open Civic Data API I was working on, chuckled
and put it aside as a really awesome hack.
Enter Docker
It wasn't until tianon connected the dots for me
and suggested a Docker FDW did I get really excited.
Cue a few hours of hacking, and I'm proud to say -- here's
Docker FDW.
This lets us ask all sorts of really interesting questions out of the API,
and might even help folks writing webapps avoid adding too much Docker-aware
logic. Abstractions can be fun!
Setting it up
I'm going to assume you have a working Multicorn, PostgreSQL and Docker setup
(including adding the postgres user to the docker group)
So, now let's pop open a psql session. Create a database (I called mine
dockerfdw, but it can be anything), and let's create some tables.
Before we create the tables, we need to let PostgreSQL know where our
objects are. This takes a name for the server, and the Python importable
path to our FDW.
Now that we have the server in place, we can tell PostgreSQL to create a table
backed by the FDW by creating a foreign table. I won't go too much into the
syntax here, but you might also note that we pass in some options - these are
passed to the constructor of the FDW, letting us set stuff like the Docker
host.
And, now that we have tables in place, we can try to learn something about the
Docker containers. Let's start with something fun - a join from containers
to images, showing all image tag names, the container names and the ip of the
container (if it has one!).
OK, let's see if we can bring this to the next level now. I finally got around
to implementing INSERT and DELETE operations, which turned out to be
pretty simple to do. Check this out:
DELETEFROMdocker_containers;
DELETE1
This will do a stop + kill after a 10 second hang behind the scenes. It's
actually a lot of fun to spawn up a container and terminate it from
PostgreSQL.
id
------------------------------------------------------------------
0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05
INSERT 0 1
(1 row)
Spawning containers works too - this is still very immature and not super
practical, but I figure while I'm showing off, I might as well go all the way.
Success! This is just a taste of what's to come, so please feel free to hack on
Docker FDW,
tweet me @paultag, file bugs / feature requests.
It's currently a bit of a hack, and it's something that I think has
long-term potential after some work goes into making sure that this is a rock
solid interface to the Docker API.
I ll be there this year!
Talks look amazing, I can t wait to hit up all the talks. Looks really well organized! Talk schedule has a bunch that I want to hit, I hope they re recorded to watch later!
If anyone s heading to PyGotham, let me know, I ll be there both days, likely floating around the talks.
I ll be giving a short talk on Debian and Docker!
I ll prepare some slides to give a brief talk about Debian and Docker, then open it up to have a normal session to talk over what Docker is and isn t, and how we can use it in Debian better.
Hope to see y all in Portland!
PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types
of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in
PKCS#11 mode using OpenSC to handle system login. 
There s a lot of things I d like to blog about. The last version of OpenStack, the OpenStack Liberty design summit, Kilo in the official jessie-backports repositories, etc. Maybe the most interesting part of this blog post is the last bit at the end, about a major change in the packaging workflow for OpenStack in Debian. Please read on
OpenStack release names reminder
Ingredients
When 
Ingredients
I know there are more of ladies in Debian, so I firstly urge you, for love of Debian, to apply if you are contributing to this project, love its community and want to see Debian taking over the universe (okay, it seems that 